Image processing device, image processing method, and program

ABSTRACT

There is provided an image processing device including a matching degree calculation unit configured to calculate a matching degree between a pixel value of a target pixel in a standard image of a current frame and a pixel value of a corresponding pixel in a reference image of the current frame, and an estimation unit configured to estimate a disparity between the standard image and the reference image based on a result obtained by calculating the matching degree. The matching degree calculation unit calculates the matching degree using a disparity estimated for the standard image and the reference image of a previous frame.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2014-018743 filed Feb. 3, 2014, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present technology relates generally to an image processing device,an image processing method, and a program. More particularly, thepresent technology relates to an image processing device, an imageprocessing method, and a program, capable of estimating a disparity in amore reliable manner.

A plurality of captured images having different viewpoints have beenused to obtain three-dimensional information in real space fordisplaying stereoscopic images. As an example, a technique forestimating disparity as three-dimensional information based on imagescaptured from left-eye and right-eye viewpoints is known (for example,refer to JP 2012-065851A).

Furthermore, when a stereoscopic image is displayed, for example, atechnique is known to compare a disparity between left-eye and right-eyeview images at time t−1 with a disparity between left-eye and right-eyeview images at time t, and then, if the change rate of disparity ishigher than a given value, to adjust the disparity at the time t (forexample, refer to JP 2011-055421A and JP 2012-178688A). This makes itpossible to prevent abrupt variation in the depth direction of an objectin a stereoscopic image, resulting in a reduction of the discomfort andfatigue of the user.

SUMMARY

The above-mentioned JP 2012-065851A obtains reliable results ofdisparity estimation in the spatial direction, but it has no descriptionon reliable results of disparity estimation in the time direction. Thus,reliable results of disparity estimation for captured images may not beobtained when the captured image has various types of noise.

To solve this, a low-pass filtering process is performed in time seriesfor results obtained by estimating disparity for each frame, and thusreliable results of disparity estimation in the time direction areobtained.

However, if a result obtained by disparity estimation for a certainframe is erroneous, the low-pass filtering process is performed based onthe erroneous result and thus the incorrect result will be provided.

The present technology is made in view of such circumstances, and it isintended to be able to estimate disparity in a more reliable manner.

According to an embodiment of the present disclosure, there is providedan image processing device including a matching degree calculation unitconfigured to calculate a matching degree between a pixel value of atarget pixel in a standard image of a current frame and a pixel value ofa corresponding pixel in a reference image of the current frame, and anestimation unit configured to estimate a disparity between the standardimage and the reference image based on a result obtained by calculatingthe matching degree. The matching degree calculation unit calculates thematching degree using a disparity estimated for the standard image andthe reference image of a previous frame.

It is possible to further provide a temporal evaluation valuecalculation unit configured to calculate a temporal evaluation valueused to evaluate a temporal variation in a disparity based on adifference between the disparity for the previous frame and thedisparity estimated for the standard image and the reference image ofthe current frame, and the matching degree calculation unit maycalculate the matching degree using the temporal evaluation value.

The temporal evaluation value calculation unit may apply a weight to thetemporal evaluation value depending on a movement in the standard imageor the reference image.

The temporal evaluation value calculation unit may set the weight to beapplied to the temporal evaluation value to be larger as a movement inthe standard image or the reference image becomes smaller.

The matching degree calculation unit may calculate the matching degree,using a pixel value of a pixel of a target region including the targetpixel in the standard image of the current frame and a pixel value of apixel of a corresponding region including the corresponding pixel in thereference image of the current frame.

It is possible to further provide a spatial evaluation value calculationunit configured to calculate a spatial evaluation value used to evaluatea spatial variation in a disparity based on a difference between adisparity estimated for a neighboring pixel located near the targetpixel and a disparity estimated for the target pixel, and the matchingdegree calculation unit may calculate the matching degree using thetemporal evaluation value and the spatial evaluation value.

The spatial evaluation value calculation unit may apply a weight to thespatial evaluation value depending on a pixel value of the target pixel.

It is possible to further provide a luminance-to-disparity conversionunit configured to convert luminance to a disparity based on a luminancevalue and a disparity for the previous frame, the luminance being aluminance value of a textureless region in the standard image of thecurrent frame, and a luminance-disparity evaluation value calculationunit configured to calculate a luminance-disparity evaluation value usedto evaluate a disparity converted from luminance based on a differencebetween the disparity converted from luminance of the standard image andthe disparity estimated for the standard image of the current frame, andthe matching degree calculation unit may calculate the matching degreeusing the temporal evaluation value and the luminance-disparityevaluation value.

The luminance-disparity evaluation value calculation unit may apply aweight to the luminance-disparity evaluation value depending onreliability of luminance-to-disparity conversion performed by theluminance-to-disparity conversion unit.

The luminance-disparity evaluation value calculation unit may set theweight to be applied to the luminance-disparity evaluation value to belarger as the reliability of luminance-to-disparity conversion performedby the luminance-to-disparity conversion unit becomes higher.

According to another embodiment of the present disclosure, there isprovided an image processing method including calculating a matchingdegree between a pixel value of a target pixel in a standard image of acurrent frame and a pixel value of a corresponding pixel in a referenceimage of the current frame, and estimating a disparity between thestandard image and the reference image based on a result obtained bycalculating the matching degree. The matching degree is calculated, inthe matching degree calculating step, using a disparity estimated forthe standard image and the reference image of a previous frame.

According to another embodiment of the present disclosure, there isprovided a program for causing a computer to execute processing ofcalculating a matching degree between a pixel value of a target pixel ina standard image of a current frame and a pixel value of a correspondingpixel in a reference image of a current frame, and estimating adisparity between the standard image and the reference image based on aresult obtained by calculating the matching degree. The matching degreeis calculated, in the matching degree calculating step, using adisparity estimated for the standard image and the reference image of aprevious frame.

In an embodiment of the present technology, a matching degree between apixel value of a target pixel in a standard image of a current frame anda pixel value of a corresponding pixel in a reference image of thecurrent frame is calculated, and a disparity between the standard imageand the reference image based on a result obtained by calculating thematching degree is estimated. In particular, the matching degree iscalculated using a disparity estimated for a standard image and areference image of a previous frame.

According to one or more embodiments of the present technology, it ispossible to estimate disparity in a more reliable manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary configuration of asystem including an image processing device according to an embodimentof the present technology;

FIG. 2 is a block diagram illustrating another exemplary configurationof the system including the image processing device according to anembodiment of the present technology;

FIG. 3 is a diagram illustrated to describe the principle of disparityestimation;

FIG. 4 is a block diagram illustrating an exemplary configuration of animage processing device known in related art;

FIG. 5 is a diagram illustrated to describe the disparity estimationperformed by the image processing device known in related art;

FIG. 6 is a block diagram illustrating an exemplary functionalconfiguration of the image processing device according to an embodimentof the present technology;

FIG. 7 is a block diagram illustrating an exemplary functionalconfiguration of a disparity estimation unit;

FIG. 8 is a flowchart illustrated to describe a disparity estimationprocess;

FIG. 9 is a block diagram illustrating another exemplary functionalconfiguration of the disparity estimation unit;

FIG. 10 is a flowchart illustrated to describe a disparity estimationprocess;

FIG. 11 is a block diagram illustrating another exemplary functionalconfiguration of the image processing device;

FIG. 12 is a block diagram illustrating still another exemplaryfunctional configuration of the disparity estimation unit;

FIG. 13 is a flowchart illustrated to describe a disparity estimationprocess; and

FIG. 14 is a block diagram illustrating an exemplary hardwareconfiguration of a computer.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Embodiments of the present technology are described below with referenceto the drawings. The description is made in the following order.

1. Configuration of image processing system

2. Principle of disparity estimation

3. Configuration of image processing device known in related art

4. Configuration of image processing device according to embodiment ofpresent technology

5. Operation of image processing device

6. Other configurations and operations of image processing device

<1. Configuration of Image Processing System>

FIG. 1 illustrates an exemplary configuration of an image processingsystem including an image processing device according to an embodimentof the present technology.

The image processing system shown in FIG. 1 is configured to include animaging device 11, an image processing device 12, a display device 13,and a storage device 14. As shown in FIG. 1, the image processing device12 is connected to the imaging device 11, the display device 13, and thestorage device 14.

The imaging device 11 generates a plurality of images captured fromdifferent viewpoint positions and supplies the images to the imageprocessing device 12 or the storage device 14. For example, the imagingdevice 11 generates an image captured from a left-eye viewpoint position(hereinafter, referred to as “left image”) and an image captured from aright-eye viewpoint position (hereinafter, referred to as “rightimage”), and supplies the images to the image processing device 12 orthe storage device 14.

The image processing device 12 estimates a disparity from a plurality ofimages captured from different viewpoint positions generated by theimaging device 11 or from a plurality of images captured from differentviewpoint positions stored in the storage device 14. The imageprocessing device 12 then supplies a result obtained by disparityestimation to the display device 13 or the storage device 14.

For example, the image processing device 12 performs disparityestimation by using left and right images which are generated by theimaging device 11 or stored in the storage device 14. The imageprocessing device 12, when performing disparity estimation using animage stored in the storage device 14, supplies a result obtained by thedisparity estimation to the storage device 14 so that the storage device14 may store the result in association with an image stored therein.

The display device 13 displays a stereoscopic image based on the imagegenerated by the imaging device 11 or stored in the storage device 14and the result of disparity estimation obtained by the image processingdevice 12.

The display device 14 stores a plurality of images captured fromdifferent viewpoint positions generated by the image device 11 or storesthe result of disparity estimation obtained by the image processingdevice 12.

The image processing system may be configured so that the imageprocessing device 12 is connected to the imaging device 11, the displaydevice 13, and the storage device 14 through a network 20 via a wired orwireless connection as shown in FIG. 2. The image processing system alsomay be configured so that the image processing device 12 is incorporatedinto any one of the imaging device 11, the display device 13, and thestorage device 14.

<2. Principle of Disparity Estimation>

The principle of disparity estimation according to an embodiment of thepresent technology is now described with reference to FIG. 3.

An embodiment of the present technology estimates disparity between aleft image L and a right image R captured respectively by two cameras(left and right cameras) as shown in FIG. 3.

In FIG. 3, optical axes of the left and right cameras are assumed to beparallel to each other. In addition, it is assumed that, when a point Pon the surface of an object in the three-dimensional space is projectedonto an image sensor of each of the left and right cameras to obtain animage, each position of the left and right cameras is correctedphysically or each of the left and right images L and R is correctedelectronically so that displacement in the vertical direction (y-axisdirection) between the left image L and the right image R at the point Pis equal to zero.

Thus, in each of the left and right images, the point P is located onthe same line (the same y-coordinate) and is deviated only in thehorizontal direction (x-axis direction). When a corresponding point(pixel) on the left image L corresponding to the point P has coordinates(x_(L),y) and a corresponding point on the right image R correspondingto the point P has coordinates (x_(R),y), the amount of deviation|x_(L)−x_(R)| in the horizontal direction (x-axis direction) is adisparity D.

In general, estimating a disparity is equivalent to calculating theabsolute value of difference between pixel values of the left image Land the right image R and obtaining a corresponding point, and thisprocess is called stereo matching.

Specifically, the disparity estimation calculates a matching degreebetween pixel values of the left image L and the right image R, which isexpressed as an evaluation formula E indicating the absolute value ofthe difference between the pixel value I_(L)(x_(L),y) of the left imageL and the pixel value I_(R)(x_(R),y) of the right image R as given inthe following Equation (1), where the left image L is a standard imageand the right image R is a reference image.E=|I _(L)(x _(L) ,y)−I _(R)(x _(L) −d,y)|  (1)

In Equation (1), a pixel (x_(L)−d,y) on the right image R in which theevaluation formula E has the minimum value is searched by changing adisparity d to be a candidate (hereinafter, referred to as “candidatedisparity d”) in a fixed range on the basis of a pixel (xL,y) on theleft image L, thereby obtaining a corresponding point. In other words, acandidate disparity d in which the evaluation formula E has the minimumvalue is the finally obtained disparity D, as given in the followingEquation (2).

$\begin{matrix}{D = {\underset{d}{argmin}(E)}} & (2)\end{matrix}$

The configuration of an image processing device known in related artwhich performs disparity estimation using the above-described methodwill be described.

<3. Configuration of Image Processing Device Known in Related Art>

FIG. 4 is a block diagram illustrating an exemplary configuration of animage processing device known in related art which performs disparityestimation.

An image processing device 31 is configured to include a disparityestimation unit 41, a frame buffer 42, and a low-pass filter 43.

The disparity estimation unit 41 performs disparity estimation for aleft image and a right image of the current frame inputted to the imageprocessing device 31 using the left image as a standard image and theright image as a reference image. The disparity estimation unit 41 thensupplies a disparity image obtained as a result by the disparityestimation to the frame buffer 42 and the low-pass filter 43.

The disparity image is an image having the disparity obtained for eachof the corresponding points of the left and right images (hereinafter,referred also to as “disparity value”) as a pixel value of each pixel ofthe disparity image. As the distance from a subject to a camera becomeslarger, the disparity value decreases. As the distance from a subject toa camera becomes smaller, the disparity value increases.

The frame buffer 42 stores a disparity image, which is supplied from thedisparity estimation unit 41, for one frame.

The low-pass filter 43 performs a low-pass filtering process on thedisparity image of the current frame supplied from the disparityestimation unit 41 using a disparity image of the immediately previousframe stored in the frame buffer 42. The disparity image subjected tothe low-pass filtering process is outputted as a final disparity imagefor the current frame.

This configuration makes it possible to obtain a reliable result of thedisparity estimation in the time direction.

FIG. 5 is a diagram illustrated to describe disparity estimationperformed by the image processing device 31 shown in FIG. 4.

As shown in the upper portion of FIG. 5, two images D_(t−1) and D_(t)indicate disparity images outputted from the disparity estimation unit41 at time t−1 and time t, respectively. In FIG. 5, the shades of colorof the disparity images are representative of the magnitude of disparityvalues. As color becomes darker, the disparity value decreases (asubject is located away from a camera). As color becomes lighter, thedisparity value increases (a subject is located near a camera).

In the example shown in FIG. 5, it is assumed that, among the subjects,two cubes are located near a camera, a column is located away from acamera, and the positional relationship between the camera and theobjects is not changed during the interval from time t−1 to time t.

It is assumed that the disparity estimation unit 41 is unable toestimate properly a disparity value of a cube at the right side in thedisparity image D_(t) in the time t due to noise or other factors, asthe right-side cube is represented by the darker color although it islocated nearer to a camera. The disparity images D_(t−1) and D_(t) areweighted and added together by the low-pass filter 43, and even in thedisparity image D_(t)′ shown in the lower portion of FIG. 5, which issubjected to the low-pass filtering process, the right-side cube isrepresented by the darker color although it is located nearer to acamera, and thus an erroneous result of disparity estimation will beobtained. The disparity image D_(t)′ is used for disparity estimation attime t+1, and thus an erroneous result of disparity estimation at time twill be provided.

The configuration of the image processing device that performs thedisparity estimation to be robust against the effect of noise or thelike is described below.

<4. Configuration of Image Processing Device According to Embodiment ofPresent Technology>

FIG. 6 is a block diagram illustrating an exemplary configuration of theimage processing device according to an embodiment of the presenttechnology.

As shown in FIG. 6, the image processing device 12 is configured toinclude a frame buffer 51, a motion detection unit 52, a disparityestimation unit 53, and a frame buffer 54.

The following description is made using a left image and a right imageas a standard image and a reference image, respectively, among imagesinputted to the image processing device 12, but the left image and theright image may be used as a reference image and a standard image,respectively.

The frame buffer 51 stores a left image (standard image), which isinputted to the image processing device 12, for one frame.

The motion detection unit 52 detects movement of a subject in a leftimage (standard image) and a right image (reference image) inputted tothe image processing device 12 and supplies motion informationindicating the detected movement to the disparity estimation unit 53.For example, the motion detection unit 52 obtains the difference betweenframes using a standard image of the current frame and a standard imageof the immediately previous frame stored in the frame buffer 51 todetect the movement of a subject. The movement of a subject may bedetected by obtaining the difference between frames using a referenceimage of the current frame and a reference image of the immediatelyprevious frame.

The disparity estimation unit 53 performs disparity estimation on astandard image and a reference image inputted to the image processingdevice 12 for each frame, supplies a disparity image obtained as aresult of the disparity estimation to the frame buffer 54, and thenoutputs the disparity image to other devices connected to the imageprocessing device 12. Specifically, the disparity estimation unit 53performs disparity estimation using the motion information supplied fromthe motion detection unit 52 and the disparity image of the immediatelyprevious frame stored in the frame buffer 54, and detailed descriptionthereof will be described later with reference to FIG. 7.

The frame buffer 54 stores the disparity image, which is supplied fromthe disparity estimation unit 53, for one frame.

An exemplary configuration of the disparity estimation unit 53 is nowdescribed in more detail with reference to FIG. 7.

As shown in FIG. 7, the disparity estimation unit 53 is configured toinclude a matching degree calculation unit 61, a temporal evaluationvalue calculation unit 62, and an estimation unit 63.

The matching degree calculation unit 61 calculates a matching degreeindicating a degree of match between pixel values (specifically,luminance value) of a target pixel to be a target in a standard image ofthe current frame and a corresponding pixel to be corresponded to thetarget pixel in a reference image of the current frame. The calculationof the matching degree is performed using a temporal evaluation valuesupplied from a temporal evaluation value calculation unit 62, which isdescribed later. The matching degree calculation unit 61 then suppliesthe calculated matching degree to the estimation unit 63.

The temporal evaluation value calculation unit 62 calculates a temporalevaluation value used to evaluate a temporal variation in disparityusing the motion information supplied from the motion detection unit 52and a disparity value corresponding to the target pixel in the disparityimage of the immediately previous frame supplied from the frame buffer54. The temporal evaluation value calculation unit 62 then supplies thecalculated temporal evaluation value to the matching degree calculationunit 61.

The estimation unit 63 estimates a disparity of the standard image andthe reference image for each pixel based on a result obtained bycalculating the matching degree by the matching degree unit 61 andoutputs a disparity image including a disparity value obtained for eachpixel.

The operation of the image processing device 12 having theabove-described configuration to perform disparity estimation is nowdescribed.

<5. Operation of Image Processing Device>

FIG. 8 is a flowchart illustrated to describe a disparity estimationprocess performed by the image processing device 12 described above. Thedisparity estimation process illustrated in FIG. 8 is a processperformed for a given single frame, and the disparity estimation processis performed every time a left image and a right image are supplied foreach frame from the imaging device 11 or the storage device 14.

In step S11, the disparity estimation unit 53 acquires a left image(standard image) and a right image (reference image) inputted to theimage processing device 12.

In step S12, the disparity estimation unit 53 acquires motioninformation supplied from the motion detection unit 52. This motioninformation indicates movement of a subject in the standard image andthe reference image.

In the following description, the process is performed by assuming thata predetermined pixel on a given line in a standard image of the currentframe is set as a target pixel.

In step S13, the disparity estimation unit 53 sets a candidate disparityd used to search a corresponding pixel in the reference image on thebasis of the target pixel in the standard image. The candidate disparityd is set to be changed within a fixed range as described above.

In step S14, the temporal evaluation value calculation unit 62 applies aweight to a disparity value D_(previous) and a value |D_(previous)−d| bya weighting factor λ_(temporal) depending on the motion informationsupplied from the motion detection unit 52 and thus calculates atemporal evaluation value λ_(temporal)|D_(previous)−d|. The disparityvalue D_(previous) is a value corresponding to a target pixel in adisparity image of the immediately previous frame. The value|D_(previous)−d| is the absolute value of the difference between thedisparity value and the candidate disparity d.

The absolute value |D_(previous)−d| is a value that represents thecontinuity of disparity in the time direction. When the change inpositional relationship between a camera and an object as a subject issmall, the absolute value |D_(previous)−d| is small. On the other hand,when the change in positional relationship between a camera and anobject as a subject is large, the value |D_(previous)−d| is large.

The temporal evaluation value calculation unit 62 controls dynamicallythe weighting factor λ_(temporal) by analyzing whether the positionalrelationship between a camera and an object as a subject is actuallychanged based on the motion information supplied from the motiondetection unit 52.

In other words, if it is determined that there is no movement betweenthe immediately previous frame and the current frame and a positionalrelationship between the camera and the object as a subject has littlevariation based on the motion information supplied from the motiondetection unit 52, the temporal evaluation value calculation unit 62increases the weighting factor λ_(temporal), resulting in an increase inthe temporal evaluation value λ_(temporal)|D_(previous)−d|.

On the other hand, if it is determined that there is any movementbetween the immediately previous frame and the current frame and apositional relationship between the camera and the object as a subjecthas a variation based on the motion information supplied from the motiondetection unit 52, the temporal evaluation value calculation unit 62decreases the weighting factor λ_(temporal), resulting in a decrease inthe temporal evaluation value λ_(temporal)|D_(previous)−d|.

Then, in step S15, the matching degree calculation unit 61 calculates amatching degree between a luminance value of a target pixel in thestandard image and a luminance value of a corresponding pixel in thereference image using the temporal evaluation value that is calculatedby the temporal evaluation value calculation unit 62.

Specifically, the matching degree calculation unit 61 sets the sum ofthe temporal evaluation value λ_(temporal)|D_(previous)−d| and theabsolute value (hereinafter, referred also to as “stereo matching term”)of the difference between a pixel value I_(L)(x_(L),y) of the targetpixel on the left image L and a pixel value I_(R)(x_(L)−d,y) of acorresponding pixel on the right image R as an evaluation formula E, asshown in the following Equation (3), and then calculates the matchingdegree between luminance values of the target pixel in the standardimage and the corresponding pixel in the reference image.E=|I _(L)(x _(L) ,y)−I _(R)(x _(L) −d,y)|+λ_(temporal) |D _(previous)−d|  (3)

As described above, when there is no movement between the immediatelyprevious frame and the current frame, the term of the temporalevaluation value λ_(temporal)|D_(previous)−d| in Equation (3) is large,and thus the effect on the evaluation formula E increases. As a result,it is possible to reduce variation in the results obtained by thedisparity estimation in the time direction. On the other hand, whenthere is any movement between the immediately previous frame and thecurrent frame, the term of the temporal evaluation valueλ_(temporal)|D_(previous)−d| in Equation (3) is small, and thus theeffect on the evaluation formula E decreases. As a result, the resultsobtained by the disparity estimation in the time direction are allowedto be varied.

In step S16, the matching degree calculation unit 61 determines whetherthe matching degree is calculated for all the candidate disparities dthat varies within a fixed range.

In step S16, if it is not determined that the matching degree iscalculated for all the candidate disparities d, then the process returnsto step S13 and the subsequent process is repeated.

On the other hand, in step S16, if it is determined that the matchingdegree is calculated for all the candidate disparities d, then theprocess proceeds to step S17. In step S17, the estimation unit 63estimates a disparity D from among the candidate disparities d byapplying the above Equation (2) to the evaluation formula E that iscalculated for each candidate disparity d by the matching degreecalculation unit 61.

In this way, a disparity value for one target pixel is obtained.

After step S17, in step S18, the disparity estimation unit 53 determineswhether a disparity is estimated for all the pixels on the line.

In step S18, if it is not determined that the disparity is estimated forall the pixels on the line, the process returns to step S13 and thesubsequent process is performed on the remaining pixels on the line.

On the other hand, in step S18, if it is determined that the disparityis estimated for all the pixels on the line, the process proceeds tostep S19. In step S19, the disparity estimation unit 53 determineswhether the disparity is estimated for all the lines.

In step S19, if it is not determined that the disparity is estimated forall the lines, then the process returns to step S13 and the subsequentprocess is performed on other lines.

On the other hand, in step S19, if it is determined that the disparityis estimated for all the lines, then the disparity estimation unit 53outputs a disparity image for one frame and then the disparityestimation process for one frame is completed.

According to the above-described process, the continuity of disparity inthe time direction is considered using a temporal evaluation value incalculating the matching degree between corresponding points of the leftand right images, and thus even when the disparity estimation for agiven frame produces an erroneous result, it is possible to estimate thedisparity in a more reliable manner without producing erroneous results.

In particular, the weighting factor for the temporal evaluation valuemay be controlled dynamically based on motion information, and thus itis possible to obtain a more reliable result of the disparity estimationin the time direction with respect to a stationary object as well as amoving object.

An image contains various types of noise, and thus the calculation ofthe matching degree by comparing pixels in the standard and referenceimages to each other is susceptible to noise, as shown by the stereomatching term of Equation (1) or (3). Accordingly, the matching degreemay be calculated by comparing pixels in a given region of the standardand reference images.

Specifically, the matching degree calculation unit 61 sets the sum ofthe sum total of the absolute value of the difference between pixelvalues of pixels of a region (target region) containing the target pixelon the left image L and a region (corresponding region) containing thecorresponding pixel on the right image R and the temporal evaluationvalue λ_(temporal)|D_(previous)−d| as an evaluation formula E, as shownin the following Equation (4), and then calculates the matching degreebetween a luminance value of the target pixel in the standard image anda luminance value of the corresponding pixel in the reference image.

$\begin{matrix}{E = {{\sum\limits_{n}{\sum\limits_{m}\left( \left| {{I_{L}\left( {{x_{L} + m},{y + n}} \right)} - {I_{R}\left( {{x_{L} + m - d},{y + n}} \right)}} \right| \right)}} + {\lambda_{temporal}{\quad\left| {D_{previous} - d} \right|}}}} & (4)\end{matrix}$

In Equation (4), a first term on the right side is the stereo matchingterm that is used to compare a luminance value of a pixel of thestandard image and a luminance value of a pixel of the reference imagein a region of (2M−1)×(2N−1) (where, M and N are positive values) on thebasis of the target pixel in the standard image. In Equation (4), m andn are values that satisfy −M<m<M and −N<n<N, respectively.

As described above, the matching degree is calculated by comparing apixel in a region of the standard image to a pixel in a region of thereference image, and thus it is possible to reduce the effect of noise,as compared to the case in which the matching degree is calculated bysimply comparing a pixel of the standard image to a pixel of thereference image.

In the above description, the weighting factor λ_(temporal) iscontrolled dynamically based on the motion information, but for example,the weighting factor λ_(temporal) may be controlled dynamically bydetermining whether a subject is a moving object using objectrecognition, or the weighting factor λ_(temporal) may be controlleddynamically by determining whether a subject is a moving object usingcolor discrimination.

The image processing device according to an embodiment of the presenttechnology may be configured to perform disparity estimation that ismore robust against the effect of noise or the like.

<6. Other Configurations and Operations of Image Processing Device>

(Reliable Disparity Estimation in Spatial Direction)

FIG. 9 is a block diagram illustrating another exemplary configurationof the disparity estimation unit 53.

In the disparity estimation unit 53 of FIG. 9, structural elements thathave a substantially similar function to that provided in the disparityestimation unit 53 of FIG. 7 are denoted with the same names andreference numerals, and repeated description thereof is omitted.

In other words, the disparity estimation unit 53 of FIG. 9 is differentfrom the disparity estimation unit 53 of FIG. 7 in that a spatialevaluation value calculation unit 71 is provided.

The spatial evaluation value calculation unit 71 calculates a spatialevaluation value for evaluating a spatial variation in disparity using adisparity (disparity value) estimated previously for a neighboring pixelthat is a pixel located near a target pixel by the estimation unit 63.The spatial evaluation value calculation unit 71 then supplies thecalculated spatial evaluation value to the matching degree calculationunit 61.

The matching degree calculation unit 61 of FIG. 9 calculates thematching degree between a target pixel of a standard image and acorresponding pixel of a reference image using the temporal evaluationvalue supplied from the temporal evaluation value calculation unit 62and the spatial evaluation value supplied from the spatial evaluationvalue calculation unit 71.

The disparity estimation process performed by the image processingdevice 12 including the disparity estimation unit 53 configured asdescribed above is now described with reference to the flowchart of FIG.10.

The process of steps S31 to S34 and S37 to S40 in the flowchart of FIG.10 is substantially similar to the process of steps S11 to S14 and S16to S19 in the flowchart of FIG. 8, and thus description thereof isomitted.

In step S35, the spatial evaluation value calculation unit 71 applies aweight to a disparity value D_(neighbor) for a neighboring pixel locatednear a target pixel and the absolute value |D_(neighbor)−d| of thedifference between the disparity value and a candidate disparity d by aweighting factor λ_(spatial) depending on a pixel value of the targetpixel, and thus calculates a spatial evaluation valueλ_(spatial)|D_(neighbor)−d|. For example when the pixel position of atarget pixel in a standard image is (x_(L),y), the neighboring pixel isa pixel located at a position (x_(L)−1,y) adjacent to the left side ofthe target pixel or a pixel located at a position (x_(L),y−1) adjacentto the upper side of the target pixel.

The absolute value |D_(neighbor)−d| is a value that represents thecontinuity of disparity in the spatial direction. If it is assumed thatthe disparity value gradually varies on the same object, when a targetpixel is located in a flat region on the same object, the absolute value|D_(neighbor)−d| is small. On the other hand, when a target pixel islocated in a boundary region between objects, the absolute value|D_(neighbor)−d| is large.

Thus, the spatial evaluation value calculation unit 71 controlsdynamically the weighting factor λ_(spatial) by analyzing whether atarget pixel is located in a flat region or in a boundary region basedon a pixel value of the target pixel in an edge image corresponding tothe standard image.

In other words, if it is determined that a pixel value of the targetpixel in the edge image is smaller than a predetermined threshold andthe target pixel is located in a flat region, then the spatialevaluation value calculation unit 71 increases the weighting factorλ_(spatial), resulting in an increase in the spatial evaluation valueλ_(spatial)|D_(neighbor)−d|.

On the other hand, if it is determined that a pixel value of the targetpixel in the edge image is greater than the predetermined threshold andthe target pixel is located in a boundary region, then the spatialevaluation value calculation unit 71 decreases the weighting factorλ_(spatial), resulting in a decrease in the spatial evaluation valueλ_(spatial)|D_(neighbor)−d|.

Then, in step S36, the matching degree calculation unit 61 calculatesthe matching degree between a luminance value of a target pixel in astandard image and a luminance value of a corresponding pixel in areference image. This calculation is performed using the temporalevaluation value calculated by the temporal evaluation value calculationunit 62 and the spatial evaluation value calculated by the spatialevaluation value calculation unit 71.

Specifically the matching degree calculation unit 61 sets the sum of thestereo matching term, the temporal evaluation valueλ_(temporal)|D_(previous)−d|, and the spatial evaluation valueλ_(spatial)|D_(neighbor)−d| as an evaluation formula E, as shown in thefollowing Equation (5), and then calculates the matching degree betweena luminance value of a target pixel in a standard image and a luminancevalue of a corresponding pixel in a reference image.

$\begin{matrix}{E = {{\sum\limits_{n}{\sum\limits_{m}\left( \left| {{I_{L}\left( {{x_{L} + m},{y + n}} \right)} - {I_{R}\left( {{x_{L} + m - d},{y + n}} \right)}} \right| \right)}} + {\lambda_{temporal}{\quad\left| {D_{previous} - d} \middle| {+ \lambda_{spatial}} \middle| {D_{neighbor} - d} \right|}}}} & (5)\end{matrix}$

As described above, when a target pixel is located in a flat region, theterm of the spatial evaluation value λ_(spatial)|D_(neighbor)−d| inEquation (5) is large, and thus the effect on the evaluation formula Eincreases. As a result, it is possible to reduce a variation in theresults obtained by the disparity estimation in the spatial direction.On the other hand, when a target pixel is located in a boundary region,the term of the spatial evaluation value λ_(spatial)|D_(neighbor)−d| inEquation (5) is small, and thus the effect on the evaluation formula Edecreases. As a result, the results obtained by the disparity estimationin the spatial direction are allowed to be varied.

According to the above-described process, the continuity in the spatialdirection is considered by using a spatial evaluation value in additionto the continuity of disparity in the time direction in calculating thematching degree between corresponding points of the left and rightimages, and thus it is possible to estimate the disparity in a morereliable manner.

For example, in a region in which light is reflected on a surgicalinstrument or the like (hereinafter, referred to as “specular reflectionregion”) in an image of a surgical scene obtained in an endoscope systemprovided with a twin-lens camera, the luminance value is large and thestandard and reference images have low correlation. Thus, the stereomatching is not performed correctly and the result of disparityestimation may be erroneous.

In this regard, the weighting factor λ_(spatial) may be controlleddynamically by analyzing whether a target pixel is located in a specularreflection region.

Specifically, if it is determined that the luminance value of a targetpixel is greater than a predetermined threshold and a target pixel islocated in a specular reflection region, then the weighting factorλ_(spatial) is set to be large. If it is determined that the luminancevalue of a target pixel is smaller than the predetermined threshold anda target pixel is not located in a specular reflection region, then theweighting factor λ_(spatial) is set to be small.

Thus, in the specular reflection region, the disparity value is notvaried significantly from a disparity value estimated for a region nearthe specular reflection region and it is possible to estimate thedisparity reliably.

In addition, even in a region with no depth in an image, that is, aregion with no texture, the stereo matching may not be performedcorrectly and the result of disparity estimation may be erroneous. Thefollowing description is given of the configuration in which thedisparity is estimated reliably even in a region with no depth.

(Reliable Disparity Estimation in Consideration of Depth)

FIG. 11 is a block diagram illustrating another exemplary configurationof the image processing device 12.

In the image processing device 12 of FIG. 11, structural elements thathave a substantially similar function to that provided in the imageprocessing device 12 of FIG. 6 are denoted with the same names andreference numerals, and repeated description of these structuralelements is omitted.

In other words, the image processing device 12 of FIG. 11 is differentfrom the image processing device 12 of FIG. 6 in that a texturelessregion disparity estimation unit 81 and a textureless region disparityvalue acquisition unit 82 are provided.

The textureless region disparity estimation unit 81 detects a regionhaving no texture (a textureless region) by performing texture analysison a standard image of the current frame, estimates a disparity for thetextureless region, and supplies the estimated result to the disparityestimation unit 53.

Specifically, the textureless region disparity estimation unit 81obtains luminance-disparity characteristics from the relationshipbetween a luminance value and disparity value in a texture-less regionof the previous frame and converts the luminance value in a texturelessregion of a standard image of the current frame to the disparity value,and thus estimates the disparity for the textureless region. A techniquethat performs disparity estimation by converting luminance values todisparity values is referred to as a disparity from luminance (DfL)disparity estimation method, and a disparity value obtained using theDfL disparity estimation method is hereinafter referred to as DfLdisparity (a DfL disparity value).

The textureless region disparity value acquisition unit 82 obtains adisparity value for the textureless region detected by the texturelessregion disparity estimation unit 81 based on the disparity imageoutputted from the disparity estimation unit 53 and supplies theobtained disparity value to the textureless region disparity estimationunit 81.

FIG. 12 is a block diagram illustrating an exemplary configuration ofthe disparity estimation unit 53 in the image processing device 12 ofFIG. 11.

In the disparity estimation unit 53 of FIG. 12, structural elements thathave a substantially similar function to that provided in the disparityestimation unit 53 of FIG. 9 are denoted with the same names andreference numerals, and repeated description of these structuralelements is omitted.

In other words, the disparity estimation unit 53 of FIG. 12 is differentfrom the disparity estimation unit 53 of FIG. 9 in that a DfL disparityevaluation value calculation unit 91 is provided.

The DfL disparity evaluation value calculation unit 91 calculates a DfLdisparity evaluation value used to evaluate a DfL disparity valueobtained by the textureless region disparity estimation unit 81 andsupplies the calculated DfL disparity evaluation value to the matchingdegree calculation unit 61.

A disparity estimation process performed by the image processing device12 configured as described above is now described with reference to theflowchart illustrated in FIG. 13.

The process of steps S51 to S55 and S58 to S61 in the flowchart of FIG.13 is substantially similar to the process of steps S31 to S35 and S37to S40 in the flowchart of FIG. 10, and thus description thereof isomitted.

In step S56, the DfL disparity evaluation value calculation unit 91applies a weight to a DfL disparity value D_(DfL) obtained for a targetpixel and the absolute value |D_(DfL)−d| of the difference between theDfL disparity value D_(DfL) and a candidate disparity d by a weightingfactor λ_(DfL) depending on the reliability of luminance-to-disparityconversion (DfL disparity estimation) performed by the texturelessregion disparity estimation unit 81, and thus calculates a DfL disparityevaluation value λ_(DfL)|D_(DfL)−d|.

The absolute value |D_(DfL)−d| is a value that represents a degree ofmatch between the DfL disparity value obtained by performing DfLdisparity estimation and the disparity value estimated by stereomatching. As the two disparity values approach to each other, theabsolute value |D_(DfL)−d| becomes small.

The DfL disparity evaluation value calculation unit 91 controlsdynamically the weighting factor λ_(DfL) based on the reliability of DfLdisparity estimation performed by the textureless region disparityestimation unit 81.

In other words, if the reliability of DfL disparity estimation performedby the textureless region disparity estimation unit 81 is determined tobe high, then the DfL disparity evaluation value calculation unit 91increases the weighting factor λ_(DfL), resulting in an increase in theDfL disparity evaluation value λ_(DfL)|D_(DfL)−d|.

On the other hand, if the reliability of DfL disparity estimationperformed by the textureless region disparity estimation unit 81 isdetermined to be low, then the DfL disparity evaluation valuecalculation unit 91 decreases the weighting factor λ_(DfL), resulting ina decrease in the DfL disparity evaluation value λ_(DfL)|D_(DfL)−d|.

Note that, when no target pixel is located in a textureless region, theweighting factor λ_(DfL) is equal to zero.

In step S57, the matching degree calculation unit 61 calculates thematching degree between a luminance value of a target pixel in astandard image and a luminance value of a corresponding pixel in areference image. This calculation is performed using the temporalevaluation value calculated by the temporal evaluation value calculationunit 62, the spatial evaluation value calculated by the spatialevaluation value calculation unit 71, and the DfL disparity evaluationvalue calculated by the DfL disparity evaluation value calculation unit91.

Specifically, the matching degree calculation unit 61 sets the sum ofthe stereo matching term, the temporal evaluation valueλ_(temporal)|D_(previous)−d|, the spatial evaluation valueλ_(spatial)|D_(neighbor)−d|, and the DfL disparity evaluation valueλ_(DfL)|D_(DfL)−d| as an evaluation formula E, as shown in the followingEquation (6), and then calculates the matching degree between aluminance value of a target pixel in a standard image and a luminancevalue of a corresponding pixel in a reference image.

$\begin{matrix}{E = {{\sum\limits_{n}{\sum\limits_{m}\left( \left| {{I_{L}\left( {{x_{L} + m},{y + n}} \right)} - {I_{R}\left( {{x_{L} + m - d},{y + n}} \right)}} \right| \right)}} + {\lambda_{temporal}{\quad\left| {D_{previous} - d} \middle| {+ \lambda_{spatial}} \middle| {D_{neighbor} - d} \middle| {+ \lambda_{Dfl}} \middle| {D_{Dfl} - d} \right|}}}} & (6)\end{matrix}$

As described above, when the reliability of DfL disparity estimation ishigh, the term of the DfL disparity evaluation value λ_(DfL)|D_(DfL)−d|in Equation (6) is large, and thus the effect on the evaluation formulaE increases. As a result, it is possible to reduce a variation in theresults obtained by the disparity estimation in the textureless region.On the other hand, when the reliability of DfL disparity estimation islow, the term of the DfL disparity evaluation value λ_(DfL)|D_(DfL)−d|in Equation (6) is small, and thus the effect on the evaluation formulaE decreases. As a result, the results obtained by the disparityestimation in the textureless region are allowed to be varied.

According to the above-described process, the presence or absence ofdepth is considered using a DfL disparity evaluation value in additionto the continuity of disparity in the time direction and the continuityof disparity in the spatial direction in calculating the matching degreebetween corresponding points of the left and right images, and thus itis possible to estimate the disparity in a more reliable manner.

The techniques of disparity estimation according to an embodiment of thepresent technology described above may be used in combination withdisparity estimation methods using dynamic programming as described inJP 2012-065851A and the like or other optimization techniques.

The above-described series of process steps may be implemented inhardware, software, or a combination of both. When the series of processsteps are implemented in software, programs that constitute suchsoftware are installed into a computer. Examples of the computer includea computer incorporated into dedicated hardware and a general-purposepersonal computer or the like that is capable of executing variousfunctions by installation of various programs.

FIG. 14 is a block diagram illustrating an exemplary hardwareconfiguration of a computer that executes the above-described series ofprocess steps according to a program.

In the computer, a central processing unit (CPU) 901, a read only memory(ROM) 902, and a random access memory (RAM) 903 are connected to eachother via a bus 904.

The bus 904 is also connected to an input/output interface 905. Theinput/output interface 905 is connected to an input unit 906, an outputunit 907, a storage unit 908, a communication unit 909, and a drive 910.

The input unit 906 includes keyboards, mice, microphones, or any otherinput devices. The output unit 907 includes displays, speakers, or anyother output devices. The storage unit 908 includes hard disk,non-volatile memory, or the like. The communication unit 909 includesnetwork interfaces or the like. The drive 910 drives a removable medium911 including magnetic disks, optical disks, magneto-optical disks,semiconductor memory or the like.

In the computer configured as described above, the CPU 901 loads aprogram that is stored, for example, in the storage unit 908 onto theRAM 903 via the input/output interface 905 and the bus 904, and executesthe program. Thus, the above-described series of process steps areperformed.

The program to be executed by the computer (CPU 901) may be providedwhile being recorded on the removable medium 911 in the form of apackaged medium or the like. The program may be provided via wired orwireless transmission media, such as a local area network, the Internet,or digital satellite broadcasting.

The computer may allow the program to be installed in the storage unit908 via the input/output interface 905 by inserting the removable medium911 into the drive 910. Further, the program may be received by thecommunication unit 909 via wired or wireless transmission media, and maybe installed in the storage unit 908. Moreover, the program may beinstalled previously in the ROM 902 or the storage unit 908.

The program executed by a computer may be a program that is processed intime series according to the sequence described in this specification ora program that is processed in parallel or at necessary timing such asupon calling.

An embodiment of the present technology is not limited to theembodiments described above, and various changes and modifications maybe made without departing from the scope of the present technology.

For example, the present technology may be embodied in cloud computingstructure in which one function is shared among devices via a network,and processing is performed by the devices cooperating with one another.

The respective steps described with reference to the above-describedflowchart may be carried out by one device or can be shared amongdevices.

In a case where multiple processes are included in one step, theprocesses included in the step may be performed by a single device orcan be shared among devices.

Additionally, the present technology may also be configured as below.

-   (1) An image processing device including:

a matching degree calculation unit configured to calculate a matchingdegree between a pixel value of a target pixel in a standard image of acurrent frame and a pixel value of a corresponding pixel in a referenceimage of the current frame; and

an estimation unit configured to estimate a disparity between thestandard image and the reference image based on a result obtained bycalculating the matching degree,

wherein the matching degree calculation unit calculates the matchingdegree using a disparity estimated for the standard image and thereference image of a previous frame.

-   (2) The image processing device according to (1), further including:

a temporal evaluation value calculation unit configured to calculate atemporal evaluation value used to evaluate a temporal variation in adisparity based on a difference between the disparity for the previousframe and the disparity estimated for the standard image and thereference image of the current frame,

wherein the matching degree calculation unit calculates the matchingdegree using the temporal evaluation value.

-   (3) The image processing device according to (2), wherein the    temporal evaluation value calculation unit applies a weight to the    temporal evaluation value depending on a movement in the standard    image or the reference image.-   (4) The image processing device according to (3), wherein the    temporal evaluation value calculation unit sets the weight to be    applied to the temporal evaluation value to be larger as a movement    in the standard image or the reference image becomes smaller.-   (5) The image processing device according to any one of (1) to (4),    wherein the matching degree calculation unit calculates the matching    degree, using a pixel value of a pixel of a target region including    the target pixel in the standard image of the current frame and a    pixel value of a pixel of a corresponding region including the    corresponding pixel in the reference image of the current frame.-   (6) The image processing device according to any one of (2) to (5),    further including:

a spatial evaluation value calculation unit configured to calculate aspatial evaluation value used to evaluate a spatial variation in adisparity based on a difference between a disparity estimated for aneighboring pixel located near the target pixel and a disparityestimated for the target pixel,

wherein the matching degree calculation unit calculates the matchingdegree using the temporal evaluation value and the spatial evaluationvalue.

-   (7) The image processing device according to (6), wherein the    spatial evaluation value calculation unit applies a weight to the    spatial evaluation value depending on a pixel value of the target    pixel.-   (8) The image processing device according to any one of (2) to (7),    further including:

a luminance-to-disparity conversion unit configured to convert luminanceto a disparity based on a luminance value and a disparity for theprevious frame, the luminance being a luminance value of a texturelessregion in the standard image of the current frame; and

a luminance-disparity evaluation value calculation unit configured tocalculate a luminance-disparity evaluation value used to evaluate adisparity converted from luminance based on a difference between thedisparity converted from luminance of the standard image and thedisparity estimated for the standard image of the current frame,

wherein the matching degree calculation unit calculates the matchingdegree using the temporal evaluation value and the luminance-disparityevaluation value.

-   (9) The image processing device according to (8), wherein the    luminance-disparity evaluation value calculation unit applies a    weight to the luminance-disparity evaluation value depending on    reliability of luminance-to-disparity conversion performed by the    luminance-to-disparity conversion unit.-   (10) The image processing device according to (9), wherein the    luminance-disparity evaluation value calculation unit sets the    weight to be applied to the luminance-disparity evaluation value to    be larger as the reliability of luminance-to-disparity conversion    performed by the luminance-to-disparity conversion unit becomes    higher.-   (11) An image processing method including:

calculating a matching degree between a pixel value of a target pixel ina standard image of a current frame and a pixel value of a correspondingpixel in a reference image of the current frame; and

estimating a disparity between the standard image and the referenceimage based on a result obtained by calculating the matching degree,

wherein the matching degree is calculated, in the matching degreecalculating step, using a disparity estimated for the standard image andthe reference image of a previous frame.

-   (12) A program for causing a computer to execute processing of:

calculating a matching degree between a pixel value of a target pixel ina standard image of a current frame and a pixel value of a correspondingpixel in a reference image of a current frame; and

estimating a disparity between the standard image and the referenceimage based on a result obtained by calculating the matching degree,

wherein the matching degree is calculated, in the matching degreecalculating step, using a disparity estimated for the standard image andthe reference image of a previous frame.

What is claimed is:
 1. An image processing device comprising: circuitryconfigured to calculate a matching degree between a pixel value of atarget pixel in a standard image of a current frame and a pixel value ofa corresponding pixel in a reference image of the current frame; andestimate a disparity between the standard image and the reference imagebased on a result obtained by calculating the matching degree, whereinthe circuitry calculates the matching degree using a disparity estimatedfor the standard image and the reference image of a previous frame, thecircuitry calculates a temporal evaluation value used to evaluate atemporal variation in a disparity based on a difference between thedisparity for the previous frame and the disparity estimated for thestandard image and the reference image of the current frame, and thecircuitry calculates the matching degree using the temporal evaluationvalue.
 2. The image processing device according to claim 1, wherein thecircuitry applies a weight to the temporal evaluation value depending ona movement in the standard image or the reference image.
 3. The imageprocessing device according to claim 2, wherein the circuitry sets theweight to be applied to the temporal evaluation value to be larger as amovement in the standard image or the reference image becomes smaller.4. The image processing device according to claim 3, wherein thecircuitry calculates the matching degree, using a pixel value of a pixelof a target region including the target pixel in the standard image ofthe current frame and a pixel value of a pixel of a corresponding regionincluding the corresponding pixel in the reference image of the currentframe.
 5. The image processing device according to claim 1, wherein thecircuitry calculates a spatial evaluation value used to evaluate aspatial variation in a disparity based on a difference between adisparity estimated for a neighboring pixel located near the targetpixel and a disparity estimated for the target pixel, and the circuitrycalculates the matching degree using the temporal evaluation value andthe spatial evaluation value.
 6. The image processing device accordingto claim 5, wherein the circuitry applies a weight to the spatialevaluation value depending on a pixel value of the target pixel.
 7. Theimage processing device according to claim 1, wherein the circuitryconverts luminance to a disparity based on a luminance value and adisparity for the previous frame, the luminance being a luminance valueof a textureless region in the standard image of the current frame, thecircuitry calculates a luminance-disparity evaluation value used toevaluate a disparity converted from luminance based on a differencebetween the disparity converted from luminance of the standard image andthe disparity estimated for the standard image of the current frame, andthe circuitry calculates the matching degree using the temporalevaluation value and the luminance-disparity evaluation value.
 8. Theimage processing device according to claim 7, wherein the circuitryapplies a weight to the luminance-disparity evaluation value dependingon reliability of luminance-to-disparity conversion performed by thecircuitry.
 9. The image processing device according to claim 8, whereinthe circuitry sets the weight to be applied to the luminance-disparityevaluation value to be larger as the reliability ofluminance-to-disparity conversion performed by the circuitry becomeshigher.
 10. An image processing method comprising: calculating, usingcircuitry, a matching degree between a pixel value of a target pixel ina standard image of a current frame and a pixel value of a correspondingpixel in a reference image of the current frame; and estimating adisparity between the standard image and the reference image based on aresult obtained by calculating the matching degree, wherein the matchingdegree is calculated, in the calculating of the matching degreecalculating, using a disparity estimated for the standard image and thereference image of a previous frame, and the method further comprisingcalculating a temporal evaluation value used to evaluate a temporalvariation in a disparity based on a difference between the disparity forthe previous frame and the disparity estimated for the standard imageand the reference image of the current frame, and calculating thematching degree using the temporal evaluation value.
 11. Anon-transitory computer readable medium including executableinstructions, which when executed by a computer cause the computer toexecute an image processing method, the method comprising: calculating amatching degree between a pixel value of a target pixel in a standardimage of a current frame and a pixel value of a corresponding pixel in areference image of a current frame; and estimating a disparity betweenthe standard image and the reference image based on a result obtained bycalculating the matching degree, wherein the matching degree iscalculated, in the calculating of the matching degree calculating, usinga disparity estimated for the standard image and the reference image ofa previous frame, and the method further comprising calculating atemporal evaluation value used to evaluate a temporal variation in adisparity based on a difference between the disparity for the previousframe and the disparity estimated for the standard image and thereference image of the current frame, and calculating the matchingdegree using the temporal evaluation value.